Aective Multimodal Interaction with a 3d Agent

The exchange of a ect/emotions is continuous in human-human interaction and plays a key role in decision making. We propose a similar paradigm for human-computer interaction and describe a novel method for classifying a user's input to a computer system for some very basic emotional attitudes. The paradigm is based on multiple communication channels including not only traditional media (eye tracker, gesture recognition, conventional \speech to text" recognition) but also on signals communicated through extra-linguistic (or \nontextual") features of the oral mode (prosodic analysis). A strategy for fusion of concordant input is described, as we believe that a ect is communicated through many channels simultaneously. Our experimental environment centres on an autonomous simpleminded dog-like 3D agent called Bouncy. Our preliminary evaluation of the agent-paradigm furnishes good grounds for believing that the proposed fusion strategy can be used also for revealing very complex attitudes like irony and certain kinds of humour realised as a contradiction between input channels (e.g. between what the user says and how he says it).